Despite its high search efficiency, differential architecture search (DARTS) often selects network architectures with dominated skip connections which lead to performance degradation. However, theoretical understandings on this issue remain absent, hindering the development of more advanced methods in a principled way. In this work, we solve this problem by theoretically analyzing the effects of various types of operations, e.g.

The authors theoretically prove that "more skip connections the faster convergence" and "shallow cells benefit faster convergence rate than deep cells". Is there any experimental evidence to verify these claims? However, does pooling operations also have a slower convergence rate than skip connections? In lines 175-178, the authors mentioned that skip connection in shared path and convolution in private path can benefit the Gram matrix singularity of networks. Thus, the convergence rate can be greatly improved.

Despite its high search efficiency, differential architecture search (DARTS) often selects network architectures with dominated skip connections which lead to performance degradation. However, theoretical understandings on this issue remain absent, hindering the development of more advanced methods in a principled way. In this work, we solve this problem by theoretically analyzing the effects of various types of operations, e.g. We prove that the architectures with more skip connections can converge faster than the other candidates, and thus are selected by DARTS. This result, for the first time, theoretically and explicitly reveals the impact of skip connections to fast network optimization and its competitive advantage over other types of operations in DARTS.